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(54) A method for automatically classifying images into events 



(57) A method for automatically classifying images 
into events, the method includes the steps of: receiving 
a plurality of images having either or both date and/or 
time of Image capture; determining one or more largest 
time differences of the plurality of images based on 
clustering of the Images; and separating the plurality of 



images into the events based on having one or more 
boundaries between events which one or more bounda- 
ries correspond to the one or more largest time differ- 
ences. 
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Description 

nELD OF THE INVENTION 

[0001] The invention relates generally to the field of s 
image processing having image understanding that 
automatically classifies pictures by events and the like 
and, more particularly, to such automatic classification 
of pictures by time and date analysis and by block- 
based analysis which selectively compares blocks of 
the images with each other. 

BACKGROUND OF THE INVENTION 

[0002] Pictorial Images are often classified by the par- 
ticular event subject or the like Ibr convenience of 
retrieving, reviewing, and albuming of the images. Typi- 
cally, this has been achieved by manually segmenting 
the images, or by the bebw-described automated 
method. The automated method includes grouping by 
color, shape or texture of the images for partitioning the 
images into groups of similar image characteristics. 
[0003] Although tiie presently known and utilized 
methods for partitioning images are satisfactory, there 
are drawbacks. The manual classification is ofcrviously 
time consuming, and the automated process, although 
theoretically classifying tiie images into events. Is sus- 
ceptible to miss-classification due to the inherent inac- 
curacies involved with classification by color, shape or 
texture. 

[0004] Consequently, a need exists for overcoming the 
above-described drawbacks. 

SUMMARY OF THE INVENTIQN 

[0005] The present invention is directed to overcoming 
one or more of the problems set forth above. Briefly 
summarized, according to one aspect of the present 
invention, the invention resides in a method for automat- 
ically classifying images into events, tiie method com- 
prising the steps of: receiving a plurality of images 
having eltiier or both date and/or time of Image capture; 
determining one or more largest time differences of the 
plurality of images based on clustering of or more larg- 
est time differences of tiie plurality of images based on 
clustering of tiie images; and separating the plurality of 
images into the events based on having one or more 
boundaries between events which one or more bounda- 
ries correspond to the one or more largest time differ- 
ences. 

[0006] These and other aspects, objects, features and 
advantages of the present invention will be more dearly 
understood and appreciated from a review of the follow- 
ing detailed description of the preferred embodiments 
and appended claims, and by reference to the accom- 
panying drawings. 



ADVANTAGEOUS EFFECT OF THE INVENTION 

[0007] The present invention has the advantage of 
improved classification of images by utilizing both date 
and time information and bk)ck-based comparison that 
checks for similarity of subject and background in tiie 
images. If date and time information is not available, 
tiien the block-based analysis may be used as the sole 
basis for classification. 

BRIEF PESCRIPTIQN OF THE PBAWINGS 
[0OO8] 

Rg. 1 1s a block diagram illustrating an overview of 
the present invention; 

Rg. 2 is a block diagram illustrating a date and time 
clustering technique of the present invention; 
Rg. 3 is a graph illustrating a scaling function used 
to map the result of the 2-means clustering; 
Rg. 4 Is a graph illustrating' a typical result of the 
scaling function of Rg. 3; 

Rg. 5 is a diagram illustrating a block diagram of an 
event boundary checking after tiie date and time 
clustering; 

Rg. 6 is a diagram illustrating grouping of images 
within each event based on content; 
Rg. 7 is a block diagram of a groupHfnerging step of 
the present invention; 

Rg. 8 is a block diagram of Image re-arrangement 
within each group; 

Rg. 9 Is a flowchart of block-based histogram corre- 
lation technique; 

Rg. 10 is diagram illustrating the conparison 
between block histogram; 
Rg. 1 1 is diagram of an example of best intersec- 
tion mapping for three segment analysis; and. 
Rg. 12 is an illustration of shift detection wHhin the 
block based histogram correlation. 

DETAILED DESCRIPTIGN OF THE INVENTION 

[0009] In tiie following description, the present inven- 
tion will be described in the prefenred embodiment as a 
software program. Those skilled in the art will readily 
recognize ttiat tiie equivalent of such software may also 
be constructed in hardware. 
[001 0] Still further, as used herein, computer readable 
storage medium may comprise, for example; magnetic 
storage media such as a magnetic disk (such as a 
floppy disl^ or magnetic tape; optical storage media 
such as an optical disc, optical tape, or machine reada- 
ble bar code; solid state electronic storage devices such 
as random access memory (RAM), or read only mem- 
ory (ROM); or any other physical device or medium 
employed to store a computer program. 
[0011] In addition, tiie term "event" is defined herein 
as a significant occun'ence or happening as perceived 
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by the subjective intent of the user of the image capture 
device. 

[001 2] Before describing the present invention, it facii- 
itates understanding to note that the present invention is 
preferably utilized on any well-known computer system, s 
such a personal computer. Consequently, the computer 
system will not be discussed in detail herein. It is also 
Instructive to note that the images are either directly 
input into the computer system (for example by a digital 
camera) or digitized before input into tiie computer sys- 
tem (for example by scanning). 
[0013] Referring to now Fig. 1. tiiere is illustrated a 
flow diagram illustrating an overview of the present 
invention. Digitized images are input into the computer 
system where a software program of the present inven- 
tion will classify them into distinct categories. The 
images will first be ranked S10 in chronological order by 
analyzing the time and date of capture of each Image. 
The date and/or time of capture of each picture may be 
extracted, for example, from the encoded information on 
the film strip of ttie Advanced Photo System (APS) 
images, or from Information available from some digital 
cameras. The representations of the images will then be 
placed 820 into one of a plurality of distinct events by a 
date and time clustering analysis that is described 
below. Within each event, the contents of ttie Images 
are analyzed S20 for determining whether images clos- 
est In time to an adjacent event should be maintained in 
the event as defined by tiie clustering analysis, or the 
adjacent events merged together. After ttie images are 
defined into events, a further sub-classrf kation (group- 
ing) within each event is performed. In this regard, the 
images witiiin each event will then be analyzed by con- 
tent S30 for grouping images of similar content together, 
and then the date and time S30 for further refining ttie 
grouping. 

[0014] The event segmentation S20 using ttie date 
and time information is by a k-means clustering tech- 
nique, as will be desaibed in detail below, which groups 
the images into events or segments. A boundary check 
Is ttien performed on the segments 320 for verifying ttiat 
the boundary images should actually be grouped into 
the segment identified by the clustering, as will also be 
desaibed below. 

[0015] These groups of Images are then sent to a 
block-based histogram correlator 830 for analyzing ttie 
content. For each event or segment sent to ttie correla- 
tor, a content or subject grouping 830 Is performed tiier- 
eon for further sub-classifying the images by subject 
witiiin the particular event segment. For example, within 
one event several different subjects may appear, and 
these subject groupings define tiiese particular sub- 
jects. The subject grouping Is based primarily on image 
content, whtoh is performed by a block-based histogram 
correlation technique. This correlation compares por- 
tions of two Images witti each other, as will also be 
described in detail below. The result of ttie ranking Is the 
classification of images of each segment into distinct 



subject groupings. The date and time of all ttie images 
within each subject grouping are then compared to 
check whettier any two or more subject grouping can be 
merged into a single subject grouping 830. 
[001 6] A refinement and subject re-arrangement anal- 
ysis 840 will further improve the overall classification 
and the subject grouping by rearranging certain images 
within a subject group. 

[001 7] Referring to Fig. 2, there is shown an exploded 
block diagram illustrating ttie data and time clustering 
technique 820. Rrst, the time Interval between adjacent 
pictures (time difference) is computed 820a. A histo- 
gram of the time differences is tiien computed S20b. an 
example of which is shown in block 10. The abscissa of 
ttie histogram is preferably the time in minutes, and the 
ordinate of ttie histogram is ttie number of pictures hav- 
ing the specified time difference. The histogram is then 
mapped 820c to a scaled histogram using a time differ- 
ence scaling function, which Is shown In Fig. 3. This 
mapping is to take the pictures with small time differ- 
ence and substantially maintain its time difference, and 
to take pictures with a large time difference and com- 
press tiielr time difference. 

[0018] A 2-means clustering is then performed 820d 
on the mapped time-difference histogram for separating 
ttie mapped histogram 1 0 into two clusters based on the 
time difference; the dashed line represents the separa- 
tion point for the two clusters. For further details of 2- 
means clustering. Introduction to Statistical Pattern 
Recognition. 2^ edition by Keinosuke Fukunaga 1990 
may be consulted, and therefore, the process of 2- 
means clustering will not be discussed in detail herein. 
Refemng briefly to Fig. 4, the result of 2-means cluster- 
ing Is the segmentation of ttie histogram into two por- 
tions 10a and 10b. Normally, events are separated by 
large time differences. The 2-means clustering, there- 
fore, is to define where these large time differences 
actually exist. In this regard, ttie right portion 10b of ttie 
2-means clustering output defines the large time differ- 
ences that correspond to ttie event boundaries. 
[001 9] Referring to Rg. 5. ttiere Is illustrated an exam- 
ple of boundary checking fc)etween events. For two con- 
secutive events i and j, a plurality of block-based, 
histogram comparisons are made to check if the pic- 
tures at tiie border of one event are different from ttie 
pictures at the border of the other event. If the compari- 
son of content is similar, the two segments are merged 
into one segment. Othenwise, tiie segments are not 
merged. Preferably, the comparisons are performed on 
the ttiree border images of each event (13, 14, 15 with j1 , 
j2, j3), as illustrated in the drawing. For example, image 
15 is compared with Image j1 and etc. This block-based 
histogram comparison technique will be described in 
detail hereinbelow. 

[0020] Refemng to Fig. 6, ttiere Is illustrated an over- 
view of subject (content) grouping for each segmented 
event. Witiiin ea<^ segmented event I. adjacent pictures 
are compared (as illustirated by ttie an^ows) witti each 
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other using the below-described, block-based histo- 
gram technique. For exannple. the block-based histo- 
gram technique may produce five subject groupings (for 
exanple groups 1-5) from the one event i. as illustrated 
In the drawing. The arrangement of the subject grouping 
Is stored for future retrieval during the subject arrange- 
ment step s40. In particular, the subject grouping having 
a single image is stored (for example groups 2, 3, and 
5). 

[0021] Refem'ng to Fig. 7, after the grouping by con- 
tent, a time and date ordering is performed on the 
groupings for merging groups together based on a time 
and date analysis. A histogram of the time difference 
between adjacent images in the event is computed, sim- 
ilar to Rg. 4. A predetermined number of the largest 
time differences (for example boundary a^2 ) com- 
pared with the boundaries (for example boundaries bi2. 
^3> ^45 ) of the subject grouping determined by the 
block-based analysis. The predetermined number of 
largest time differences are determined by dividing the 
total number of images within an event by the average 
number of picture per group (four Is used In the present 
invention). If the boundary of the subject grouping 
matches the boundary based on the chosen time differ- 
ences, the subject groupings will not be merged. If there 
Is not a match between the two kx>undarles, the subject 
groupings having a boundary not having a matched 
time difference in the histogram will be merged into a 
single subject grouping (for example groups bi, bb, bs 
merged into resulting group c^). 
[0022] Referring to Rg. 8. there is illustrated a diagram 
of Image re-arrangement within each groip. The 
arrangement of the initial subject groupings is retrieved 
for identifying subject groupings that contain single 
images (for example the groups with a single image of 
Fig. 6 -groups 2. 3, and 5 that are re-illustrated as 
groups 2, 3, and 5 in Fig. 8). Any single Images from the 
same sut^ect grouping that are merged as identified by 
the merged subject grouping (for example, groups c^ 
and C2 of Rg. 7) are compared with all other images in 
the merged subject grouping, as illustrated by the 
arrows. This comparison is based on block-based histo- 
gram analysis. If the comparisons are similar, these 
images will be re-an^anged so that the similar images 
are located adjacent each other, for example groups d^ 
and 62- 

[0023] Further refinement Is done by comparing any 
group that still contains a single image after the above 
procedure, with all the images In the event. This is to 
check if these single image groups can be better 
arranged within the event grouping. This comparison is 
similar to the subject re-arrangement step of Fig. 8. 
[0024] Referring to Fig. 9. there is illustrated a flow- 
chart of the block-based histogram con-elation used in 
the above analyses. First, a histogram of the entire 
image of both images is computed S50. a global histo- 
gram. A comparison of the two histograms is performed 
by histogram intersection value S60 illustrated the fol- 



lowing equation: 

n 

2;min(fl„C, 

« Inter (f?. C) = «=2— j; 

10 where R Is the histogram of the reference image, C Is 
the histogram of the candidate image, and n is the 
number of bins in the histogram. If the intersection is 
under a threshold S65, preferably 0.34. atthough other 
thresholds may be used, the images are different. If the 

IS threshold Is met or exceeded S65. then a block-based 
histogram correlation will be performed S7Q. In this 
regard, each image will be divided into blocks of a given 
size, preferably 32 x 32 pixels in the present invention. It 
is instructive to note that those skilled In the art may 

20 vary the block size depending on the resolution of the 
image without departing from the scope of the invention. 
For each blocK a color histogram Is computed. Refer- 
ring to Fig. 10. if one image is considered a reference 
image and one image a candidate image, the images 

25 are compared in the fbllowing way. Each block 20 of the 
reference Image Is compared to the oon'esponding 
block 30 of the candidate Image and to the adjacent 
blocks 40. 8 blocks in the present invention. 
[0025] Referring to Fig. 9. the block histograms 

30 between the reference Image and the candkJate Image 
are compared using the histogram Intersection equation 
defined above S80. The average Intersection value Is 
derived by computing the average of the best intersec- 
tion values from each of the block comparisons S90. 

35 This average intersection value will be compared to a 
low threshokJ (preferably 0.355). and a high threshold 
(preferably 0.557). If the average Intersection value is 
below the low threshold S95. the two Images are con- 
sidered different. If the average Intersection value is 

40 above the high threshold S96, then the two images are 
considered similar. If the average intersection value is 
between these two thresholds, further analysis will t>6 
performed as described below (3-segment average 
intersection map SI 00). 

45 [0026] Referring to both Rgs. 9 and 1 1 , a 3-segment 
analysis will be performed to determine if the two 
images may contain a similar subject. This is performed 
by first forming a map 60 which contains the average of 
the two highest Intersectbn values of each of the block 

50 comparisons; for example. 9 comparisons were per- 
formed in the illustration of Fig. 10. the average of the 
highest two will be used for map 60. Fig. 1 1 Illustrates, 
for example, a 9 x 6 block although it should be under- 
stood that the map size depends on the size of the 

55 image. This map is divided into three parts: the left por- 
tion 70a, the center portion 70b. and the right portion 
70c. If the average intersection value of the center por- 
tion 70b is higher than a threshold (preferably 0.38) 
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SI 05. the two images may contain a very similar subject 
in the center portion 70b of the image, and the two 
images may be considered to be similar by subject. In 
addition, the comparisons of the histogram will be per* 
formed with the reference and candidate images s 
reversed. If the two images are similar both methods 
should give substantially similar correlation; obviously if 
they are different, the results will not be similar. The 
images are then checked Si 10 to determine if there is a 
high intersection value in one of the directions, right, io 
left, up, and down. 

[0027] Referring to Rgs. 9 and 12, shift detection is 
used to determine the case when the two images 90 
and 100 (of two different sizes in ttie drawing) have very 
similar subject that appears in different locations of the is 
image. For example, the main subject may be situated 
in the center of one image and to the left-hand side of 
the other image. Such a shift can be determined by 
recording both the best intersection values of the refer- 
ence blocks, as well as the coordinates of the oorre- 20 
sponding candidate blocks. This Is achieved by 
comparing the intersection values of the blocks in four 
directions (right, left, up, and down). The entire image 
will be shifted by one block (as illustrated by the arrows) 
in one of the directions (right in the drawing) where the ^ 
best intersection value is the highest. The above analy- 
sis and the shift can be repeated Sl20to check for sim- 
ilarity. 

[0028] The invention has been described with refer- 
ence to a prefen-ed emkx)diment. However, it will be so 
appreciated that variations and nrxxiifications can be 
effected by a person of ordinary skill in the art without 
departing from the scope of the Inventton. 



performing the 2-means clustering. 

4. The method as in daim 2, wherein step (c) includes 
checking the images adjacent the one or vnore 
boundaries for similarity by comparing content of 
the images. 

5. The method as in claim 4. wherein step (c) includes 
checking the images adjacent the one or wore 
boundaries for similarity by using a block-based his- 
togram correlation technique. 

6. The mettiod as in daim 5 further comprising step 
(d) dividing the events into subject grouping by 
using an image content analysis. 

7. The method as in claim 6, wherein step (d) includes 
dividing the events into subject grouping by using a 
block-based histogram technique. 

8. A method for automatically dassifying images into 
events, the method comprising the steps of: 

(a) receiving a plurality of images an-anged in 
chronological order; 

(b) dividing the images into a plurality of blocks; 
and, 

(c) grouping ttie images into subject grouping 
based on block-based histogram correlation 
which includes computing a color histogram of 
each block and computing a histogram inter- 
section value which determines the similarity 
between blocks. 



Claims 

1 . A method for automatically dassifying images into 
events, the method comprising the steps of: 

(a) receiving a plurality of images having either 
or both date and/or time of image capture: 

(b) determining one or more largest time differ- 
ences of the plurality of images based on dus- 
tering of the images; and. 

(c) separating the plurality of images into the 
events teased on having one or more bounda- 
ries between events which one or more bound- 
aries correspond to the one or more largest 
time differences. 

2. The mettnod as in claim 1 , wherein step (b) includes 
computing a time difference histogram and per- 
forming a 2-means clustering on the time difference 
histogram for defining the one or more boundaries. 

3. The method as in claim 2, wherein step (b) furttier 
includes mapping the time difference histogram 
through a time difference scaling function before 



9. The metiiod as in claim 8, wherein step (c) indudes 
comparisons of two of ttie images by shiifting one of 
the images in a desired direction based on the Inter- 
section value and then computing the block based 

correlation. 

10. The mettiod as in claim 9, wherein step (c) indudes 
forming a map that contains two best intersection 

. values of each of the block comparisons; dividing 
the map into three portions: and then comparing 
center portions for similarity. 
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